begrudge (someone) the time

Translations
reprocher à quelqu'un le temps
Definitions
To feel resentful or envious about someone else's opportunity or ability to spend time on something, often implying that one believes that person does not deserve that time.